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Abstract 

Classroom assessment techniques (CATs) or other closure activities are widely promoted 
for use in college classrooms. However, research on whether CATs improve student 
learning are mixed. The authors posit that the results are mixed because CATs were de¬ 
signed to “help teachers find out what students are learning in the classroom and how 
well students are learning it” (Anglo & Cross, 1993, p.4) rather than to help students 
learn the material. Thus, in order for CATs to be effective for helping students learn, they 
should be redesigned to focus on student retention of concepts. The Ticket to Retention 2 
(TtR2) is designed to do just that. 
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The Ticket to Retention (TtR) was designed to increase student retention of concepts. The 
TtR is similar to classroom assessment techniques (CATs) because it involves students in 
writing about information covered during a class session, can be used to assess student 
understanding of concepts, and helps to determine if the professor’s teaching methodolo¬ 
gy was successful in conveying information to the students (Angelo & Cross, 1993; Cross 
& Angelo, 1988; Barkley, Cross, & Major, 2005; Weaver & Cotrell, 1985). CATs or oth¬ 
er closure activities are widely promoted for use in college classrooms (Angelo & Cross, 
1993; Barkley, 2010; Blumberg, 2009; Cross, 1998; Cross & Angelo, 1988; McGlynn, 
2001; Nilson, 2010; Richlin, 1998; Richlin, 2006; Steadman & Svinicki, 1998; Weaver & 
Cotrell, 1985; Wlodkowski, 2008). Anglo and Cross (1993) state that “Classroom As¬ 
sessment is an approach designed to help teachers find out what students are learning in 
the classroom and how well students are learning it” (p.4). 

However, the TtR was designed to do more. The original TtR combined “the benefits of 
the ticket to leave, the one minute paper, half-sheet response, the post-write strategy, and 
think-pair-share strategies” (Divoll & Browning, 2010). Although the TtR and other 
CATs are similar, the TtR differs because it is designed to use three to five focused ques¬ 
tions and peer interaction to increase students’ retention. Research on CATs found in¬ 
creased student satisfaction when such strategies are used (Cottell, 1991; Anglo & Cross, 
1993; Olmstead, 1991; Walker, 1991); however, the effect of CATs on student learning 
has interpreted as mixed (Anglo & Cross, 1993) and lacking evidence (Simpson-Beck, 
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2011). For example, Shelton (1991) and Walker (1991) found that student learning is in¬ 
creased with the use of classroom assessment, but Cottell and Hardwood (1998), Kelly 
(1991), Olmstead (1991), Simpson-Beck (2011), and Stetson (1991) found no evidence of 
increased learning. In addition, much of the research on CATs effect on learning comes 
from student survey data (i.e., Byon, 2005; Goldstein, 2007; Fabry, Eisenbach, Curry, & 
Golich, 1997; Kelly, 1991; Rouseff-Baker & Holm, 2004; Soetaert, 1998; Steadman, 
1998). The goal of designing the TtR and the Ticket to Retention 2 (TtR2) was to create a 
CAT that could result in the same benefits for professors (i.e. evaluating teaching, deter¬ 
mining student’s understanding of concepts, and increased students’ satisfaction with a 
course) (Cottell, 1991; Anglo & Cross, 1993; Olmstead, 1991; Walker, 1991), but also 
increase student learning. 

In earlier research, the TtR was used to reinforce concepts, and students retained approx¬ 
imately 88% of the information that that was reviewed with the TtR (Divoll & Browning, 
2010; Divoll, Browning, & Vesey, 2012). Despite these benefits, students who used the 
TtR seemed to score about the same on multiple choice questions at the end of the semes¬ 
ter as their peers who did not use the TtR, whereas students who used the TtR scored 
much higher on short answer questions at the midpoint of the semester than their peers 
(Divoll & Browning, 2010; Divoll, Browning, & Vesey, 2012). 

The original TtR was basically rote rehearsal since students were simply reviewing the 
information multiple times in a short amount of time, therefore the rote rehearsal strategy 
may not lead to increased retention (Sousa, 2001). Considering the literature on learning 
theory, recommendations from students who used the TtR, and in an attempt to improve 
student retention from results of the original TtR, the ticket to retention was redesigned 
into the TtR2. The TtR2 has students taking notes on the TtR2 sheets during class rather 
than only writing their answer at the end of class. Since highlighting key points during 
class helps students understand what is important and helps students learn (Bransford, 
Brown, & Cockering, 2000; Davis, 2009), the TtR2 includes the professor alerting the 
students to the key points during instruction and encourages students to write the answer 
to the questions on the TtR2 sheet when the information is discussed. 

After being alerted to the important points and writing the answers during class, social 
interaction occurs when students discuss their answers with two other students (one at a 
time). Situating learning in a social context has been shown to result in increased learning 
(Kafai & Resnick, 1996) and is recommended when teaching adults (Blumberg, 2009; 
Bransford, et. al., 2000; Davis, 2009; Donovan, Bransford, & Pellegrino, 1999; Johnson, 
Johnson, & Smith 1998; Richlin, 2006; Lawler, 1991). The TtR2 also added an aspect of 
higher level thinking into the original TtR, i.e., comparing and contrasting, which is ab¬ 
sent in the original version. In the TtR2, the students compare and contract their answers 
to the questions with that of their peers and include ideas that were discussed during the 
peer interaction if the peer’s answer added to their answer. The assumption is that by in¬ 
cluding aspects of higher level thinking, which is at the analysis level of Bloom’s taxon¬ 
omy (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956) and one of McKeachie’s (2002) 
categories for challenging questions, the TtR2 would further increase student retention. 
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Students conclude their discussion of the topics by writing the best possible answer, 
which is also something that is unique to the TtR2. 

The TtR2 is designed to improve students’ retention of concepts. Learning and retention 
are not the same (Sousa, 2001). Retention is defined as “the process whereby long-term 
memory preserves a learning in such a way that it can locate, identify, and retrieve it ac¬ 
curately in the future” (Sousa, 2001, p. 85). Barkley (2010) states, “Retention is influ¬ 
enced by many factors, but a critical one is adequate time to process and reprocess infor¬ 
mation so that it can be transferred from short-term to long-term memory” (p. 23). The 
TtR2 attempts to create adequate time using the strategy throughout the entire class ses¬ 
sion and reprocessing by using multiple-sense and multiple-method instruction, which 
results in students learning “better and more” (Nilson, 2010, p.237). 

Having students listen, write, say, and compare/ contrast the concepts results in students 
interacting with the ideas “multiple times and in different ways” (Nilson, 2010, p.237). 
Diversification in learning activities improves learning by involving multiple parts of the 
brain (Kress, Jewitt, Ogborn, & Charalampos, 2006; Verkiri, 2002). The TtR2 also pro¬ 
vides students with the feedback, practice, and review needed for students to retain in¬ 
formation (Bransford et al., 2000; Davis, 2009; Donovan et al., 1999), and allows stu¬ 
dents to reexamine concepts (Cross, 1981). 

As is learner-centered teaching, the focus of the TtR2 is to shift the role of professors 
from provider of knowledge to facilitators of learning (Barr & Tagg, 1995; Blumburg, 
2009; Wlodkowski, 2008) and to concentrate more on student learning (Blumberg, 2009; 
DeZure, 2000; Fink, 2003; Gardiner, 1994; Weimer, 2002). Although this shift seems 
logical to many, it is not the norm in the lecture-dominated college classroom (Bligh, 
2000; Lammers & Murphy, 2002). However, the shift in teaching strategies should occur 
since millennial students “want to know we care about them” (Nilson, 2010, p.12). The 
TtR2 also includes active learning in the classroom, which can result in improved student 
learning and is recommended at the college level (Flint, Zakos, & Frey, 2002; Ginsberg, 
2010; Ginsberg & Wlodkowski, 2009; Grubb,1999; Grubb & Byrd, 1999; Hackathorn, 
Solomon, Blankmeyer, Tennial, & Garczynski, 2011; Kuh, Kinzie, Schuh, Whitt, & As¬ 
sociates, 2005; McGlynn, 2001; Meyers & Jones,1993; Sorcinelli, 1991; Wlodkowski, 
2008; Woolfolk, 2011). 

Research on the TtR was based on using an assessment at the midpoint of the semester 
and at the end of the semester to determine if using the TtR resulted in higher scores 
(Divoll & Browning, 2010; Divoll, Browning, & Vesey, 2012). However, a limitation to 
the earlier research was that students were not given a pretest at the beginning of the se¬ 
mester to determine what the students knew going into the semester. This study adds to 
the growing body of literature on the TtR and TtR2 by using a pretest to determine how 
much students knew at the beginning of the semester and the impact of the TtR2 on stu¬ 
dents’ retention of concepts. In addition, this study provides further evidence that CATs 
can be enhanced to improve student retention. 
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Methodology 

Seventy undergraduate students in three sections of an education course participated in 
this mixed methods study. The setting included two four-week summer courses and one 
15-week fall undergraduate college level classroom management course at a southern 
university. Professor A taught one section of the course in the summer (section one) and 
one in the fall summer (section three). Professor B taught one course in the summer (sec¬ 
tion two). The professors, who are the authors of this paper, created the TtR2. The two 
four-week summer semester included 15 two-hour 50 minute sessions Monday through 
Thursday and the 15-week fall semester included meeting weekly for two hours and 50 
minutes. 

In each course, the students were divided into two groups, a group of students who re¬ 
ceived the TtR2 the first half of the semester (group A) and a group of students who re¬ 
ceived the TtR2 the second half of the semester (group B). Students were assigned to the 
groups randomly. The group not using the TtR2 for the week left class 10 minutes early, 
while the TtR2 group stayed to complete the TtR2. Group A participated in the TtR2 in 
classes 2-6. Both groups A and B participated in the TtR2 in class seven due to an odd 
number of classes in which the TtR2 was implemented. During class seven, a midpoint 
retention assessment (MRA), which consisted of short answer questions, was adminis¬ 
tered to determine the information retained by group A and group B. The TtR2 for class 
seven was conducted the same week as the MRA, but after the assessment for the first 
part of the semester was administered. In addition to week seven, group B participated in 
the TtR2 in classes 8, 9, 11, 12, and 13. Therefore, when the MRA was administered, 
group A was tested on the material that was reinforced using the TtR2, while group B 
was tested on the same material, but had not been given the TtR2 to reinforce the con¬ 
cepts. 

A comprehensive retention assessment (CRA) was administered during class 15 and in¬ 
cluded 49 multiple choice questions testing students’ recall of the information from the 
entire semester. Questions 1-22 were from classes 2-6, questions 23-27 were from class 
7, and questions 28-49 were from classes 8, 9, 11, 12, and 13. Thus, questions 1-27 repre¬ 
sented times when group A used the TtR2 and questions 23-49 represented times when 
group B used the TtR2. 

Data Collection and Analysis 

Data collection methods included: (a) a pretest, (b) the MRA, (c) the CRA, and (d) a stu¬ 
dent questionnaire about the TtR2. The pretest, which was administered the first class, 
was comprised of 49 multiple choice questions and assessed the students’ knowledge of 
the major concepts for the semester. The MRA and the CRA assessed only concepts that 
were included on the TtR2. The MRA was administered on the seventh week, while the 
CRA and student questionnaire were conducted during the final class (the fifteenth class). 
Each of the assessments was used to determine students’ retention of concepts taught and 
did not count as a grade. Thus, students were not given advanced notice of the MRA and 
CRA. 
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The pretest and assessment data were analyzed by assessment types and sections. The 
pretest results were compared to the MRA and CRA when the TtR2 was implemented 
and when it was not. These comparisons resulted in the creation of a change score (i.e., 
MRA minus the pretest or CRA minus the pretest) with the intervention and one without 
(the difference between the pretest and an assessment). The mean and standard deviation 
were generated for each data set. Overall, the average of all of the sections were generat¬ 
ed for the pretest, MRA, CRA, and change score. In addition, an overall average for each 
of the aforementioned data was generated when the intervention was implemented and 
again when it was not. Each calculation was rounded to the nearest tenth. 

The Mann-Whitney U statistic and the Wilcoxon statistic were used to determine if the 
distribution of scores for the individual sections significantly differed. However, these 
tests could not be run for the averages of all three sections because of the different varia¬ 
bles (professor and semester length). These tests could not be run to determine if the 
CRA and pretest with and without the TtR2 significantly differed because the students 
from groups A and B used the TtR2 with different questions (i.e., 1-27 for group A and 
23-49 for group B). 

In addition to the students’ retention assessments, the students were asked their views 
about the TtR2. The students were asked three open ended questions about the TtR2 (68 
out of the 70 students complete the questionnaire; a 97% response rate): (a) did you learn 
more as a result of the TtR2, (b) would you use the TtR2 in your future classroom, and 
(c) general comments about the TtR2. The questionnaire data was analyzed by question 
topic using open coding to create categories and patterns. In accordance with qualitative 
methodology, after the categories and patterns were generated, a constant comparative 
approach resulted in discovering evidence for the categories and patterns across the ques¬ 
tions and axial coding was applied to each data source to make connections between the 
categories and patterns (Creswell, 1998; Rossman & Rallis, 2003). Rather than share 
each students’ comments for each pattern, the students’ comments are either summarized 
or only a few examples are given. 


Results 

Quantitative data were organized by assessment (MRA, CRA, and pretest), change score, 
section, and finally a summary of the assessments across the sections was discussed. The 
qualitative data were organized by themes that emerged across all of the sections. The 
questions on the MRA corresponded to questions 1-22 on the pretest and CRA. Questions 
one through 22 were the concepts for which only group A used the TtR2 on the CRA, 
questions 28-49 were concepts that only group B used the TtR2, and questions 23 
through 27 were concepts for which the TtR2 was used by both groups. The aforemen¬ 
tioned questions on the CRA correspond to the same questions on the pretest. 

Pretest Scores 

Sections one, two, and three averaged 68 (n = 27), 60 (n = 26), and 64 (n = 17), respec¬ 
tively, on the pretest (see Table 1). The pretest average across all sections was 64 (see 
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Table 2) with group A scoring 64 and group B averaging 64.1. Scores for groups A and B 
on the information that they were to use the TtR2 was 62.2 with the TtR2 and 66.2 with¬ 
out. The pretest average for questions one through 22 was 61.5, questions 23 through 27 
averaged 44, and questions 28-49 averaged 71.6. The Mann-Whitney U was conducted 
for each class to determine if the distribution of scores on the Pretest for questions 1-22 
and questions 28-49 significantly differed between groups (group A and B). The results 
of the Mann-Whitney U for sections one, two, and three indicated no significant differ¬ 
ence between groups for questions 1-22 and 28-49 (see Tables 3, 4, and 5). Thus, stu¬ 
dents in group A performed about the same on the pretest for questions 1-22 and 28-49. 
The data for questions 23-27 was not run for any of the sections because both groups 
used the TtR2 for these questions. 

MRA Results 

Sections one, two, and three averaged 84.6 (n = 14), 84.3 (n = 13), and 81.6 (n = 8), re¬ 
spectively (see Table 1), on the MRA with the TtR2 and 67.87 (n= 13), 69.3 (n = 13), 
66.4 (n = 9) without the TtR2. The MRA average for all three sections when the TtR2 
was used was 83.8 and 68 when it was not (15.8 point difference) (see Table 1). The 
Mann-Whitney U was conducted for each class to determine if the distribution of scores 


Table 1. Section Averages. 


Assessment 



Sections 




1 

SD 

2 

SD 

3 

SD 

Pretest 

68.0% 

9.35 

60.0% 

6.91 

64.0% 

8.78 

Pretest Group A 

69.2% 

9.21 

58.2% 

7.34 

64.0% 

7.79 

Pretest Group B 

66.6% 

9.66 

61.7% 

6.24 

64.0% 

10.05 

Pretest with TtR2 

66.8% 

12.64 

56.4% 

7.86 

63.8% 

11.75 

Pretest without TtR2 

69.0% 

14.87 

64.7% 

12.08 

63.9% 

12.20 

Pretest Ql-22 

59.1% 

10.09 

66.8% 

9.60 

57.2% 

12.56 

Pretest Q23-27 

68.9% 

16.01 

7.7% 

11.42 

60.0% 

18.71 

Pretest Q28-49 

76.3% 

13.48 

67.1% 

20.63 

71.1% 

10.53 

MRA 

76.5% 

13.37 

76.8% 

12.78 

75.9% 

16.00 

MRA with TtR2 

84.6% 

9.35 

84.3% 

11.35 

81.6% 

4.62 

MRA without TtR2 

67.8% 

11.54 

69.3% 

9.46 

66.4% 

19.29 

MRA Change Score with TtR2 

25.6% 

7.26 

26.1% 

10.67 

24.2% 

13.49 

MRA Change Score without 
TtR2 

8.7% 

11.18 

7.6% 

11.55 

9.4% 

17.42 

CRA 

88.0% 

8.24 

81.2% 

6.46 

84.4% 

7.72 

CRA with TtR2 

88.3% 

8.88 

85.5% 

7.48 

81.7% 

8.43 

CRA without TtR2 

87.2% 

10.63 

79.6% 

8.46 

80.8% 

7.62 

CRA Change Score with TtR2 

21.5% 

8.90 

29.1% 

10.56 

17.9% 

12.91 

CRA Change Score without 
TtR2 

18.2% 

13.40 

14.8% 

10.25 

16.6% 

13.61 


The Journal of Effective Teaching, Vol. 12, No. 2, 2012, 45-64 

© 

2012 All rights reserved 



The Ticket To Retention: A Classroom Assessment 


51 


Table 2. Overall Averages: Sections. 


Assessment 



Mean 

With TtR2 

Without TtR2 

Difference 

Pretest 

64.0% 

62.2% 

66.2% 

-4.0% 

Pretest Group A 

64.0% 

X 

X 

X 

Pretest Group B 

64.2% 

X 

X 

X 

MRA 

75.9% 

83.8% 

68.0% 

15.8% 

MRA Change Score 

17.0% 

25.4% 

8.5% 

17.0% 

CRA 

84.4% 

85.7% 

82.8% 

2.9% 

CRA Change Score 

20.5% 

23.4% 

16.6% 

6.8% 


on the MRA significantly differed between groups (i.e., with TtR2 vs. without TtR2) (see 
Tables 3, 4, and 5). The results for students in group A, the group who received TtR2 
during weeks 2-6, scored significantly higher on the MRA than students who did not 
receive the TtR2 in sections one and two. Despite the 15.1 point difference between 
groups A and B, the results of the Mann-Whitney U for section three indicated no signifi¬ 
cant difference between groups. 

CRA Results 

Students across the sections averaged an 84.4 on the CRA (see Table 2). The CRA aver¬ 
age for all three sections when the TtR2 was used was 85.7 and 82.8 without the TtR2. 
Sections one, two, and three averaged 88.3, 85.5, and 81.7, respectively, on the CRA with 
the TtR2 and 87.2, 79.6, and 80.8 without the TtR2 (see Table 1). 

Change Scores 

The pretest was compared to the MRA and CRA to determine what the students retained 
during the course. The difference between the pretest and the two assessments show what 
the students knew at the beginning of the semester and what knowledge they retained at 
the middle of the semester (MRA change score) and at the end of the semester from the 
course (CRA change score). We refer to the difference between the pretest and the MRA 
and CRA as a change score. 

MRA change score. Across all three sections students who used the TtR2 on the MRA 
had a change score of 25.4 points, while students who did not use the TtR2 had a change 
score of 8.5 points (see Table 2). This resulted in the students who used the TtR2 having 
a change score that was 17 points higher than their peers. The MRA change scores for 
sections one, two, and three averaged 25.6 points, 26.1 points, and 24.2 points, respec¬ 
tively, with the TtR2 and 8.7 points, 7.6 points, and 9.4 points without the TtR2. The 
Mann-Whitney U was conducted for each class to determine if the distribution of MRA 
change scores significantly differed between groups (i.e., with TtR2 vs. without TtR2) 
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Table 3. Statistical Analysis. 


Comparison 

Results 

Statistics 

P value 

Pretest Q1-Q22 (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -.418, p = .676 

Pretest Q28-Q49 (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -.984, p = .325 

MRA (TtR2 vs. no TtR2) 

S 

Mann Whitney U 

z = -3.283, p = .001 

MRA CS (TtR2 vs. no TtR2) 

s 

Mann Whitney U 

z = -3.786, p <.001 

CRA CS (TtR2 vs. no TtR2) 

NS 

Wilcoxon 

z = -1.057, p = .290 


Note. “NS” indicates that the results are non-significant, while “S” indicates significant. 


Table 4. Section 2 Statistical Analysis. 


Comparison 

Results 

Statistics 

P value 

Pretest Q1-Q22 (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -1.722, p = .085 

Pretest Q28-Q49 (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -1.142,p = .253 

MRA (TtR2 vs. no TtR2) 

S 

Mann Whitney U 

z = -3.214, p= .001 

MRA CS (TtR2 vs. no TtR2) 

s 

Mann Whitney U 

z = -3.181, p = . 001 

CRA CS (TtR2 vs. no TtR2) 

s 

Wilcoxon 

z = -3.645, p < .001 


Note. “NS” indicates that the results are non-significant, while “S” indicates significant. 


(see Tables 3, 4, and 5). The results for section one and two indicated that there was a 
significant difference between groups. Thus, students in group A, the group who received 
TtR2 during weeks 2-6, had significantly higher MRA change scores than students who 
did not receive the TtR2. Despite the 14.8 point change score for section three the results 
of the Mann-Whitney U indicated that the groups scored about the same. 

CRA change score. Across all three classes students who used the TtR2 on the CRA had 
a change score of 23.4 points, while students who did not use the TtR2 had a change 
score of 16.6 points (see Table 2). This resulted in the students who used the TtR2 having 
a change score that was 6.8 points higher than their peers. The CRA change scores for 
sections one, two, and three averaged 21.5 points, 29.10 points, and 17.9 points, respec¬ 
tively, with the TtR2 and 18.2 points, 14.9 points, and 16.8 points without the TtR2 (see 
Table 1). The Wilcoxon statistic was conducted to determine if the distribution of CRA 
change scores was significantly differed between groups (i.e., with TtR2 vs. without 
TtR2) (see Table 3, 4, 5). Students in group A, the group who received TtR2 during the 
first part of the semester performed about the same on their CRA change scores as stu¬ 
dents who did not receive the TtR2 in both sections one and three. The results from sec¬ 
tion two indicated that there was a significant difference between the two groups. 
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Table 5. Section 3 Statistical Analysis. 


Comparison 

Results 

Statistics 

P value 

Pretest Q1-Q22 (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -.097, p = .923 

Pretest Q28-Q49 (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -.537, p = .606 

MRA (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -1.742, p = .082 

MRA CS (TtR2 vs. no TtR2) 

NS 

Mann Whitney U 

z = -1.78, p = .075 

CRA CS (TtR2 vs. no TtR2) 

NS 

Wilcoxon 

z = -.166, p = .868 


Note. “NS” indicates that the results are non-significant, while “S” indicates significant. 


Student Questionnaire Data 

In addition to the students’ retention assessments, the students were asked their views 
about the TtR2. The students were asked three open ended questions about the TtR2. 

Using the TtR2 in their future classroom. Thirty-four of the 68 students (50%) suggest¬ 
ed that they would use the TtR2 in their future classrooms. Most of these 34 students, 
suggested that they would use the TtR2 in their future classroom because they believed 
their students would benefit from the TtR2. Examples of these responses included: (a) “It 
offers the students with another opportunity to hear and discuss the material and is a great 
closing for the lesson” (S2-10B) and (b) “Yes, if the ticket to retention worked for me 
and many of the other students (I heard) it will most likely work with children rather than 
just lecturing” (S2-17B). In addition, one of the students suggested that the TtR2 is a 
good strategy for non-native English speakers, “Yes, I would like to use the ticket to re¬ 
tention because as a second language learner, I learned a lot from the material and I think 
this strategy will help others students that are learning English to become success in 
class” (S3-15B). 

Twenty-four of the students (35%) indicated they may use the TtR2 in their future class¬ 
room. Of the 24 students who suggested they may use the TtR2, ten students mentioned 
that their decision was dependent on the age of students that they taught. Seven of the 
students who suggested that they may use the TtR2 in their classroom indicated that they 
might make changes to the TtR2 before using the strategy. These students suggested that 
they would either use an aspect of the TtR2: “I might use part of it such as talking to a 
partner about one aspect of the lesson” (S2-22B) or diversify the TtR2: “Possibly for 
some topics and in some forms, but I will probably change it up a little to give some vari¬ 
ety (S1-5B). 

Ten of the students (15%) indicated that they would not use the TtR2 in their classrooms. 
Of these students, four suggested that they would not use the TtR2 because they believed 
the TtR2 would not work for the age that they were teaching. The remaining six students 
mentioned that they would not use the TtR2 because they either did not see the value in 
it, they did not enjoy it and thus would not use the strategy in their classroom, or they 
would rather use other strategies. 
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Students’ perception of the TtR2 on their learning. Forty-three of the students (or 
63.2%) stated that the TtR2 had a positive impact on their learning. For example, the stu¬ 
dents suggested that the TtR2 them remember difficult concepts, being more confident 
with the information and recalling information quicker. 

Student S3-13B inferred that the TtR2 was more helpful than simply taking notes, “Yes, 
because even though I usually took notes it is not the same when you hear the answers 
twice and then rewrite them.” Another student indicated that the TtR2 resulted in learning 
the vocabulary better, “I do feel like I remembered more vocabulary from the parts that I 
did the ticket. I normally do not remember vocabulary” (S1-22B). The TtR2 seems to en¬ 
courage students who might not take notes to do so, “I knew all the questions that I did 
with the ticket to retention and the other questions I wasn't 100% sure on because I lis¬ 
tened in class but I didn’t always take notes” (S1-10A). In addition, a student who had the 
ticket the second part of the semester felt lost on the MRA because they did not use the 
TtR2, 


When I did the ticket to retention it helped me to understand better about the sub¬ 
ject or material that the teacher was teaching. Also, doing the "final exam" I was 
more secure about the materials that I learn during the semester. When I took the 
"midterm exam" I was a little lost because I did not have something that can help 
me to reinforce the material that I was learning at the beginning of the semester 
(S3-17B). 

Finally, student S1-9A posited that the TtR2 made him more aware of what the profes¬ 
sor’s focus was, “I was in the first group of ticket to retention and it made me more aware 
of what the instructor was focusing on, and because I did the ticket of retention I felt 
comfortable in taking notes afterward.” 

Twenty-three of the students (33.8%) believed that the TtR2 had little or no impact on 
their ability to recall the information on the CRA. A theme that emerged from the stu¬ 
dents who believed that the TtR2 resulted in little or no impact on their retention was that 
the use of other teaching strategies negated the potential impact of the TtR2. Students be¬ 
lieved that the other class strategies (i.e., discussion and active learning strategies) were 
just as effective as the TtR2: “I'm an auditory kinesthetic learner and with the interactive 
environment of the class I was able to retain pretty well... It all seemed about the same to 
me. The actual hands on is what helped me retain most of the information” (S1-5B). Stu¬ 
dent S3-16B, in addition to feeling that other strategies help her in class also expressed 
that she became anxious as a result of doing the TtR2, “I feel that I learned more when 
we worked together with a group... I think the TtR2 gave me anxiety more than anything 
because I would have to rush at the end of class to get it finished.” 

Five of the students who felt the TtR2 had little or no impact on their recall indicated that 
their own abilities as students helped them more than the TtR2. Student S2-7A mentioned 
that her interest in the subject was a factor in the TtR2 not resulting in improved learning, 
“I feel I remembered close to the same, because I was genuinely interested in the materi- 
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al.” Studying outside of the classroom was another factor why students did not see the 
TtR2 as something that helped them, 

I paid attention no matter if I had the ticket or not. And when I am trying to re¬ 
member something I have to go over many times - talking, hearing, and writing 
about it so going over it one time after the lecture doesn't really help me. (S3-6A) 

Another student stated that because she was an overachiever she learned regardless of the 
TtR2, “Coming from the overachiever, I find that the CRA was very easy for me because 
I paid attention for the whole class and remembered everything, so to be honest, I am not 
sure if the ticket actually helped” (S1-24B). Finally, one student suggested that she can 
leam without the TtR2 because of the type of learner she is, “I think it was about the 
same. I guess because of my type of learning is why I can retain different things” (Al- 
8B). Three students suggested that the reason the TtR2 did not result in more recall was 
because they did the TtR2 during the first part of the year. For example, student SI- 27B 
rationalized that timing of using the TtR2 was relative to their recall, “The first part of the 
CRA was from the MRA and I only felt unsure about a couple, but I think it was because 
the information was older - not because of the ticket” (S1-19B). 

Recommendations for the TtR2. Five students suggested that the TtR2 could be im¬ 
proved by starting the ticket to retention earlier so that students would not have to rush. 
One interesting theme that emerged from the data was that the TtR2 was best suited for 
weaker students. One student mentioned that the TtR2 is only good for some learners, 
“Ticket to retention is good for some learners, but also not good for others” (SI-3A). 
Students S2-7B suggested that the TtR2 would help students who are not interested in the 
topic, “I think for those who did not have an interest in the topic it may have helped a lot 
for them.” Furthermore, student S3-12B indicated that the TtR2 would be assist students 
if they did not know how to take notes, “Ticket to retention is a great strategy for helping 
students learn how to take notes.” 

A few students stated that they would have liked to have the TtR2 all semester. For ex¬ 
ample, student S2-15B posited, “I think it’s a good way to learn about the lessons. Maybe 
all students should do it all semester, instead of dividing it up. Then we can learn more 
for the whole semester versus just half.” Furthermore, two other students actually thanked 
their professor for using the TtR2 in their comments. The students stated: (a) [Professor’s 
name], my comment about the ticket to retention- it helped me to understand better the 
subject and material during the class. It was like reinforcing while I was writing, reading, 
and talking to my peers. Thank you...” (S3-15B) and (b) “I think it was a great tool, and 
the results really gave me a lot to think about. Thank you for the invaluable tool and in¬ 
formation. I really enjoyed this class” (S3-23B). 

Discussion 

This study investigated the effectiveness of the TtR2, which is an attempt to incorporate 
addition aspects into CATs so that student retention of concepts results. The study repre¬ 
sents the first time the combination of a pretest and posttest has been used to determine 
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the effectiveness of the TtR2. Previous research on the TtR (Divoll & Browning, 2010) 
tested the retention of students when they used the TtR and used a comparison group to 
determine if the TtR was effective (Divoll, Browning, & Vesey, 2012). Those studies in¬ 
dicated that students who used the TtR recalled approximately 88% of the information 
that was reviewed using the TtR Divoll & Browning, 2010) and the results of the TtR 
seemed to be more effective for short answer questions than multiple choice questions 
(Divoll, Browning, & Vesey, 2012). We theorized that the TtR2 would result in higher 
averages than the TtR since the TtR2 was redesigned from its original version to include 
aspects of teaching adults that are deemed beneficial such as highlighting key points dur¬ 
ing instruction (Bransford et. al., 2000; Davis, 2009), incorporating higher level thinking 
skills, and providing feedback, practice, and review (Bransford et al., 2000; Davis, 2009; 
Donovan, et al., 1999). 

Summary ofMRA and CRA Results 

The average for students across all three sections when tested on the information that was 
reinforced using the TtR2 on the MRA was 83.8%. This average was 15.8 points higher 
than their peers in group B. The 15.8 point difference was slightly higher than the 13 
point difference between group A and group B observed in earlier research on the TtR 
(Divoll, Browning, & Vesey, 2012). While pleased that students in the current study in 
group A scored so much higher than group B, we expected more than a 2-3 point differ¬ 
ence between results from the TtR2 and the TtR. However, not having a pretest in the 
previous study, students prior knowledge was not known before the semester began. The 
MRA scores with the TtR2 versus without the TtR2 for two of three sections were statis¬ 
tically significant. The third section was not statically significantly higher, meaning that 
the difference could have occurred by chance, but students with the TtR2 did average 
15.1 points higher than group B. In addition, section three only had 8 students who used 
the TtR2 on the MRA and 9 who did not, thus the low numbers could have impacted the 
results of the Mann-Whitney U. These results indicate that the TtR2 was successful in 
producing higher class averages on short answer tests for students who used the TtR2. 
This data suggests that CATs can be adapted to improve students’ concept retention, 
whereas previous research on the link between classroom assessment and learning was 
mixed (Anglo & Cross, 1993; Kelly, 1991; Olmstead, 1991; Shelton, 1991; Stetson, 
1991; Walker, 1991). 

When tested using multiple choice questions (CRA), the difference between information 
that was reviewed using the TtR2 and the information for which the TtR2 was not used 
resulted in a smaller increase, just under three points (85.7 to 82.8). The same results oc¬ 
curred in an earlier study on the TtR. However since the numbers where low in the previ¬ 
ous study (i.e., 21 students), the assumption was that with the redesigned TtR and the in¬ 
creased numbers (i.e., 70 students) the results of the current study would be higher. In 
this study, the difference between the TtR2 results on the MRA and the CRA could re¬ 
sulted from one of five possible reasons: (a) the comparison of two groups who had dif¬ 
ferent lengths of time between use of the TtR2 and the CRA, (b) the types of test ques¬ 
tions (i.e., short answer versus multiple choice), (c) the possibility that students in one 
group had more prior knowledge of the content that was tested on the CRA, (d) the stu- 
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dents in groups A and B used the TtR2 for different concepts on which they were as¬ 
sessed on the CRA (i.e., questions 1-27 for group A and questions 23-49 for group B) 
and/or (e) the students in one group were better students. 

One limitation of this study was that group A used the TtR2 on the first part of the semes¬ 
ter (classes 2-7), while students in group B used the TtR2 weeks (7-9 and 11-13). There¬ 
fore, time between using the TtR2 and the CRA was longer for the students in group A 
than the students in group B. For example, the information for group A that was reviewed 
using the TtR2 was anywhere from four weeks (summer semester) to thirteen weeks (fall 
semester) prior to the CRA, while the information that was not reviewed using the TtR2 
was anywhere from three days (summer semester) to seven weeks (fall semester). Thus, 
the only time the results were compared from the same time period with and without the 
intervention was on the MRA. The CRA compared different questions and length of time 
between the intervention and the assessment. This issue with the study design most likely 
contributed to discrepancy between the impact the TtR2 had on the CRA and MRA. 

Another potential explanation for the vast difference for the result on the MRA and the 
CRA was the type of questions that were used on the MRA (short answer) versus the 
CRA (multiple choice). Students tend to score better on tests when there are answer 
choices from which to choose versus when they provide an answer (short answer) (Biggs, 
1999; Carvalho, 2009; Dansereau, 1985; Gay, 1980; Nickerson, 1989). Some of the ef¬ 
fects of the TtR2 could be negated on the CRA because multiple choice questions are 
easier to answer than short answer questions (Carvalho, 2009). If the TtR2 results in im¬ 
proved recall and automaticity of recalling information, as some students suggested, then 
having the answers provided through multiple choice items may reduce the effects. Our 
earlier research resulted in the TtR having a larger effect on the MRA than the CRA 
(Divoll, Browning, & Vesey, 2012). Thus, our current study was designed to determine 
what the students knew at the beginning of the semester (i.e., pretest) and to compare stu¬ 
dent scores to what they recalled on the MRA and CRA (i.e., change score) in hope that 
this question would be answered. 

Summary of Change Scores 

As mentioned above, the pretest helped highlight what students could recall from the be¬ 
ginning of the year to the MRA and CRA. Students in group A and group B across all of 
the sections averaged about the same going into the semester 64.% (group A) to 64.2% 
(group B). The Mann-Whitney U test revealed that students in group A receiving the 
TtR2 during weeks 2-6 performed about the same on the pretest as students in group B. 
In addition, the pretest average for the information that was reviewed using the TtR2 was 
66.2, while the information that was not reviewed using the TtR2 was 64.1. The Mann- 
Whitney U statistic that was run for all three classes indicated that there was no statistical 
difference between the students in group A and B in each class on questions 1-22 and 
questions 28-49 on the pretest. The possible conclusion for the discrepancy between the 
impact of the TtR2 on the MRA and CRA is that group A happened to be better students 
than group B or that they knew more coming into the semester does not seem to be plau¬ 
sible given the aforementioned data. 
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The change score from the pretest to the MRA for the group who had the TtR2 (group A) 
averaged 25.4 points, while students who did not use the TtR2 (group B) averaged a 
change score of 8.5 point (a difference of 17). The Mann-Whitney U revealed that the 
change scores for group A in two out of the three sections significantly differed, meaning 
that the intervention was most likely responsible for the difference in those classes. Ac¬ 
cording to the Mann-Whitney U, the 15.1 difference between students who used the TtR2 
and those who did not in section three, the class that were there was no significantly dif¬ 
ference, could have resulted by chance. These results could have been influenced because 
of the now numbers (eight students in group A and nine in group B). The change scores 
between groups A and B on the MRA suggest that the TtR2 made a difference (signifi¬ 
cant difference for two out of three classes and a numerical difference for all three), espe¬ 
cially since the pretest yielded no significant difference between groups A and B at the 
beginning of the semester. 

The change score from the pretest to CRA across all three sections was 23.4 points when 
students used the TtR2, while students who did not use the TtR2 had a change score of 
16.6 points. The Wilcoxon revealed that the change scores for group A in two out of the 
three sections did not significantly differ, meaning that the difference for the two classes 
was most likely caused by chance. Across all three sections, the students who used the 
TtR2 had a change score that was 6.8 points higher than their peers. Once again, it seems 
logical that the difference between the change scores on the MRA and CRA can be at¬ 
tributed to the study design (i.e., attempting to compare two groups who had a different 
length of time between the use of the TtR2 and the CRA and the students in groups A and 
B used the TtR2 for different concepts on which they were assessed on the CRA) and/or 
the type of questions on the CRA (multiple choice) (Biggs, 1999; Carvalho, 2009; 
Dansereau, 1985; Gay, 1980; Nickerson, 1989). 

Student Questionnaire Data 

The student questionnaire data provided the students’ voice to the numerical data collect¬ 
ed for the TtR2. It is interesting that 50% of the students surveyed suggested that they 
would use the TtR2 in their future classroom and another 35% suggested that they might 
use the TtR2 in their classroom. This indicates that 85% of the students recognize the 
value of the TtR2. Only 15% of the students suggested that they would not use the TtR2 
in their future classrooms. Of that 15%, four of these 10 students suggested that they 
would not use the TtR2 because they were going to teach very young children. Thus, 
some of the students who suggested that they would not use the TtR2 in their future class¬ 
room still recognized the benefits of the TtR2, but felt young children, because of their 
limitations with reading and writing, would not benefit from the TtR2. 

Students’ perception of the TtR2 on their learning. Forty-three of the students (63.2%) 
stated that the TtR2 had a positive impact on their learning. Some students felt that the 
TtR2 resulted in recall that was more automatic, helped them recall more difficult infor¬ 
mation, improved their content vocabulary, and gave them confidence with the material, 
which are aspects that were not measured by the assessments. 
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Additionally, two students thanked their professor in their response to the questionnaire 
because they believed that theTtR2 helped them and one suggested that the TtR2 helped 
her as an English language learner. The TtR2 can be considered an effect strategy for 
English language learners because it includes: (a) one on one social interaction, (b) re¬ 
viewing and rehearsal of concepts and vocabulary, (c) using multiple senses, (d) expand¬ 
ing content vocabulary, (e) practicing listening, speaking, and writing in a meaningful 
way, (f) clarifying concepts and helps students become confident with the content, and 
(g) using active engagement (Colombo, 2012; Echevarria & Graves, 2011; Echevarria, 
Vogt, & Short, 2008; Genesee, Lindholm-Leary, Saunders, & Christian, 2006; Ginsberg 
& Wlodkowski, 2009; Levine & McCloskey, 2009; Peregoy & Boyle, 2005; Smiley & 
Salsberry, 2007; Vogt & Echevarria, 2007). The students’ comments along with the 
quantitative data above provide growing evidence that the TtR2 is effective and that 
CATs can be adapted to improve student retention. Even without the previously men¬ 
tioned data that supports the TtR2, the fact that so many students believed that the TtR2 
worked for them could result in a placebo effect wherein students are more confident and 
thus perform better. 

Of the students who suggested that the TtR2 had little or no impact on their learning, five 
suggested that the professors’ active learning strategies helped them just as much or more 
than using the TtR2. These comments also highlight another limitation of this study. This 
course, regardless of professor or semester, included many active learning activities to 
help students learn the course material. Thus, some of the recall that students had on the 
MRA and CRA could have been due to the active learning strategies since such strategies 
improve learning (Flint et al., 2002; Ginsberg & Wlodkowski, 2009; Grubb, 1999; Grubb 
& Byrd, 1999; Kuh et al, 2005; McGlynn, 2001; Meyers & Jones,1993; Sorcinelli, 1991; 
Wlodkowski, 2008; Woolfolk, 2011). Consequently, the use of active learning strategies 
could have been part of the reason for the discrepancy between the results with the TtR2 
and without the TtR2 on the MRA and CRA. It would be interesting to see how the TtR2 
impacted retention in a predominately lecture class. 

Five students also suggested that the TtR2 had little or no impact on them because they 
understand how to study and take notes, they are over achievers, and/or they were “genu¬ 
inely interested in the topic.” These students also suggested that the TtR2 should be used 
for students who do not have these skills or interest in the subject. The students raised an 
interesting point. Could part of the reason that the TtR2 was more effective for some stu¬ 
dents and not as effective for other students be a result of the students’ abilities as learn¬ 
ers, study time outside of class, learning style, and/or interest in the topic? Students who 
review the material on their own, as some students suggested, and take good notes might 
not need the TtR2 and for these students the TtR2 might not be as effective. Another is¬ 
sue mentioned by students was that the TtR2 was rushed at the end of class, often went 
past class time, and students not involved in the TtR2 that week could leave early. These 
aspects, as a student indicated, resulted in some students rushing to finish the TtR2 and 
potentially not being as invested in the process. 
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Conclusion 

CATs or other closure activities are widely promoted for use in college classrooms (An¬ 
gelo & Cross, 1993; Barkley, 2010; Blumberg, 2009; Cross, 1998; Cross & Angelo, 
1988; McGlynn, 2001; Nilson, 2010; Richlin, 1998; Richlin, 2006; Steadman & Svinicki, 
1998; Weaver & Cotrell, 1985; Wlodkowski, 2008). However, research on whether 
CATs improve student learning are mixed (Anglo & Cross, 1993; Kelly, 1991; Olmstead, 
1991; Shelton, 1991; Stetson, 1991; Walker, 1991) and therefore one can argue that the 
evidence that CATs improve learning is lacking (Simpson-Beck, 2011). The authors posit 
that the results are considered mixed or lacking because CATs were designed to “help 
teachers find out what students are learning in the classroom and how well students are 
learning it” (Anglo & Cross, 1993, p.4) rather than to help students learn the material. 
Thus, in order for CATs to be effective for helping students learn, they should be rede¬ 
signed to focus on student retention of concepts. The TtR2 is designed to do just that. 

The data reported indicate that the TtR2 is an effective method to assist students with the 
retention of concepts when tested using short answer questions. However, as was the case 
with earlier research, the results are reduced when students are retested using multiple 
choice questions (Divoll, Browning, & Vesey, 2012). The reduced effect of the TtR2 on 
multiple choice questions could have resulted because of aspects of the study design, the 
active learning strategies that were implemented in class, and the multiple choice ques¬ 
tions that were used on the CRA. Despite these results, the fact that students score better 
on short answer questions (MRA) than multiple choice questions suggests that students 
who used the TtR2 are not only able to recall the answers, but are also have enough un¬ 
derstanding to explain their answers in writing. In addition, the students’ comments show 
that the TtR2 resulted in: remembering difficult concepts, more confidence with the con¬ 
cepts, quicker recall, learning vocabulary, taking notes for some students who do not 
normally take notes, and being more aware of the professor’s key points. These are as¬ 
pects that were not measured by the assessments. 

Furthermore, when students’ prior knowledge is included in the study, the data suggest 
that the TtR2 had a positive effect on the short answer questions. The change scores for 
students (i.e., the difference between the pretest and the MRA or CRA) who used the 
TtR2 on the MRA were 17 points higher than their peers, while students who used the 
TtR2 on the CRA scored 6.8 points higher. Thus, based on what students knew at the be¬ 
ginning of the semester, the students with the TtR2 were able to recall more than their 
peers. However, the Mann-Whitney U and the Wilcoxon revealed that the change scores 
on the MRA for students with the TtR2 and without were only significant for two out of 
the three courses and one out of the three courses on the CRA. 

Although this data adds to the literature on CATs and the TtR2, further research is need¬ 
ed. Such investigations should include addressing the following questions: (a) Are the 
effects of the TtR2 more pronounced for students who have weak study skills; (b) Is the 
TtR2 a good strategy for English language learners at the college level; (c) How would 
the TtR2 perform if it was implemented in a lecture only class; and (d) Is the TtR2 only 
effective on short answer questions? 
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In addition, future research is needed that has students using and not using the TtR2 dur¬ 
ing the same weeks, and tests students in multiple types of questions. Given the impres¬ 
sive results of the TtR2 on the MRA, the limited results on the CRA, and the fact that al¬ 
most 78% of the students believed the TtR2 helped their learning, the TtR2 shows poten¬ 
tial for adapting CATs to improve student retention of concepts. 
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